Extent operation processing

ABSTRACT

Successful storing of extent operations into corresponding records of a transaction log results in acknowledgement of completion of the extent operations being indicated to one or more hosts. In response to determining that the extent operations are unrelated to each other, the extent operations are flushed in parallel from the transaction log to back-end non-volatile data storage. During the flushing, dependencies between the extent operations and other operations stored in the transaction log are maintained. Dependency chains are identified within the transaction log, and at least one tree data structure representing the dependencies between each of the extent operations and the other operations stored in the transaction log may be generated and traversed in order to select the correct operations stored in the transaction log to flush.

TECHNICAL FIELD

The present disclosure relates generally to intelligent data storagesystems that process operations performed on extents of non-volatiledata storage (“extent operations”), and more specifically to technologyfor processing extent operations using a transaction log, with parallelflushing of unrelated extent operations from the transaction log.

BACKGROUND

Data storage systems are arrangements of hardware and software that mayinclude one or more storage processors coupled to non-volatile datastorage drives, such as solid state drives and/or magnetic disk drives.Each storage processor may service host I/O requests received fromphysical and/or virtual host machines (“hosts”). The host I/O requestsreceived by the storage processor may specify one or more storageobjects (e.g. logical units (“LUNs”), and/or files, etc.) that arehosted by the storage system and store user data that is written and/orread by the hosts. Each storage processor executes software thatprocesses host I/O requests and performs various data processing tasksto organize and secure the user data that is received from the hosts andstored in the non-volatile data storage drives of the data storagesystem.

While processing host I/O requests that read and write user data storedin the non-volatile data storage of the data storage system, some datastorage systems perform operations on units of non-volatile data storagereferred to as “extents.” Each extent may be a logically contiguousportion of a shared logical address space, and may consist of and/orinclude at least a portion of a data storage volume or logical drive. Anextent operation is an operation that is performed on one or moreextents. For example, creation of a point in time copy of a unit of data(sometimes referred to as creating a “snapshot” of the data) may beperformed as an extent operation, e.g. an extent copy operation from aparent extent to a child extent that provides a point in time copy ofthe parent extent. Another example of an extent operation may bedeletion of the point in time copy, which may be performed as an extentdelete operation. Other specific operations may also be performed on orusing one or more extents.

SUMMARY

Processing of individual extent operations by previous data storagesystems has been performed completely in-line, e.g. by one or moreforeground processes, prior to indicating the completion of the extentoperations to a requesting host. Such foreground, in-line processing ofextent operations has at times resulted in significant delays before thesuccessful completion of the extent operations and/or other relatedoperations could be acknowledged to the host.

To address the above described and/or other shortcomings of previoustechnology, new technology is disclosed herein that stores multipleextent operations into corresponding records of a transaction log, andthen indicates successful completion of each of the extent operations toone or more hosts in response to the successful storage of the extentoperations into the corresponding records of the transaction log.

In another aspect, the disclosed technology determines that the multipleextent operations stored in the transaction log are unrelated to eachother. In response to determining that the extent operations areunrelated to each other, the extent operations are flushed in parallelfrom the transaction log to back-end non-volatile data storage.

In another aspect, the disclosed technology maintains dependenciesbetween the multiple extent operations and one or more other operationsthat are also stored in the transaction log while flushing the multipleextent operations in parallel from the transaction log to the back-endnon-volatile data storage.

In another aspect, the disclosed technology maintains the dependenciesbetween the extent operations and the other operations that are alsostored in the transaction log at least in part by identifying dependencychains within the transaction log. Each one of the dependency chains mayinclude one of the extent operations, and each one of the dependencychains may represent dependencies between the extent operation in thedependency chain and at least one other operation stored in thetransaction log.

In another aspect, the disclosed technology maintains the dependenciesbetween multiple extent operations and the other operations stored inthe transaction log at least in part by generating at least one treedata structure representing the dependencies between each of the extentoperations and other operations stored in the transaction log.

In another aspect, the disclosed technology further maintains thedependencies between the multiple extent operations and other operationsstored in the transaction log further at least in part by traversing theat least one tree data structure while flushing the plurality of extentoperations in parallel from the transaction log to the back-endnon-volatile data storage in order to select the operations in thetransaction log to flush.

Embodiments of the disclosed technology may provide significantadvantages over previous technical solutions. For example, because anextent operation may be considered committed after the extent operationis successfully stored into a record of the transaction log, theacknowledgement of completion of the extent operation can be morerapidly conveyed back to a host that issued the operation, e.g. bytransmitting an acknowledgement message back to the host that issued arequest that caused the extent operation to be performed. In addition,the improved parallelization provided by the disclosed technology duringthe flushing of operations from the transaction log may improve I/Operformance and resource utilization within the data storage system.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the disclosed technology will beapparent from the following description of embodiments, as illustratedin the accompanying drawings in which like reference numbers refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the disclosed technology.

FIG. 1 is a block diagram showing an example of a data storage system inwhich an example of the disclosed technology is embodied;

FIG. 2 is a block diagram showing an example of a portion of atransaction log including a dependency chain;

FIG. 3 is a block diagram showing an example of a portion of atransaction log including a dependency chain;

FIG. 4 is a block diagram showing an example of a portion of atransaction log including multiple dependency chains;

FIG. 5 is a block diagram showing an example of transaction log datastructures in some embodiments;

FIG. 6 is a flow chart showing an example of steps performed in someembodiments to perform classification;

FIG. 7 is a block diagram showing an example representation of a LogSequence Number (LSN) tree;

FIG. 8 is a block diagram showing another example representation of anLSN tree; and

FIG. 9 is a block diagram showing a further example representation of anLSN tree.

DETAILED DESCRIPTION

Embodiments of the invention will now be described with reference to thefigures. The embodiments described herein are provided only as examples,in order to illustrate various features and principles of the disclosedtechnology, and the invention is broader than the specific embodimentsdescribed herein.

Embodiments of the technology disclosed herein may provide improvementsover previous technologies by rapidly acknowledging completion of extentoperations to hosts, and increasing parallelization during flushing ofoperations from a transaction log. In the disclosed technology,indications of successful completion of individual extent operations areprovided to one or more hosts in response to the extent operations beingsuccessfully stored into corresponding records of a transaction log. Theextent operations may include copy extent and delete extent operations.In response to a determination that multiple extent operations areunrelated to each other, the extent operations may be flushed inparallel from the transaction log to back-end non-volatile data storage.Dependencies are maintained between the multiple extent operations andone or more other operations that are also stored in the transaction logwhile the multiple extent operations are flushed in parallel from thetransaction log to back-end non-volatile data storage, at least in partby identifying dependency chains within the transaction log. At leastone tree data structure may be generated to represent the dependenciesbetween the extent operations and the other operations stored in thetransaction log. The tree may be traversed while the multiple extentoperations are flushed in parallel from the transaction log to theback-end non-volatile data storage, in order to select operations to beflushed.

FIG. 1 is a block diagram showing an operational environment for thedisclosed technology, including an example of a data storage system inwhich the disclosed technology is embodied. FIG. 1 includes some numberof physical and/or virtual Host Computing Devices 110, referred to as“hosts” and shown for purposes of illustration by Hosts 110(1) through110(N). The hosts access data storage provided by Data Storage System116, for example over one or more networks, such as a local area network(LAN), and/or a wide area network (WAN) such as the Internet, etc., andshown for purposes of illustration in FIG. 1 by Network 114. DataStorage System 116 includes at least one Storage Processor 120 that iscommunicably coupled to Network 114, and Backend Non-Volatile DataStorage Drives 128, e.g. though one or more Communication Interfaces122. No particular hardware configuration is required, and StorageProcessor 120 may be embodied as any specific type of device that iscapable of processing host input/output (I/O) requests received fromHosts 110 (e.g. I/O read and I/O write requests, etc.) and persistentlystoring data received from the Hosts 110.

The Backend Non-Volatile Data Storage Drives 128 may include physicaldata storage drives such as solid state drives, magnetic disk drives,hybrid drives, optical drives, and/or other specific types of drives.Backend Non-Volatile Data Storage Drives 128 may be directly physicallyconnected to and/or contained within Storage Processor 120, and/or maybe communicably connected to Storage Processor 120 by way of one or morenetworks.

A Memory/Local Persistent Data Storage 126 in Storage Processor 120stores program code that is executable on Processing Circuitry 124.Memory 126 may include volatile memory, e.g. DRAM. Memory/LocalPersistent Data Storage 126 may also include one or more persistentphysical data storage devices that are physically separate andindependent from Backend Non-Volatile Data Storage Drives 128. Suchpersistent data storage devices in Memory/Local Persistent Data Storage126 may include a battery-backed memory (e.g. a battery-backed RAM(Random Access Memory)) that is contained in and/or communicably coupledto Storage Processor 120, and/or some other specific type of persistentdata storage. The contents of the persistent data storage devices inMemory/Local Persistent Data Storage 126 may is preserved when power toData Storage System 116 is shut off. In some embodiments, the total datastorage capacity of the persistent data storage devices in Memory/LocalPersistent Data Storage 126 may be less than the total data storagecapacity of Backend Non-Volatile Data Storage Drives 128.

Processing Circuitry 124 may, for example, include or consist of one ormore microprocessors, e.g. central processing units (CPUs), multi-coreprocessors, chips, and/or assemblies, and associated circuitry.Processing Circuitry 124 and Memory/Local Persistent Data Storage 126together form control circuitry, which is configured and arranged tocarry out various methods and functions as described herein. TheMemory/Local Persistent Data Storage 126 stores a variety of softwarecomponents that may be provided in the form of executable program code.For example, Memory/Local Persistent Data Storage 126 may store softwarecomponents such as Host I/O Processing Logic 135. When program codestored in Memory/Local Persistent Data Storage 126 is executed byProcessing Circuitry 124, Processing Circuitry 124 is caused to carryout the operations of the software components. Although certain softwarecomponents are shown in the Figures and described herein for purposes ofillustration and explanation, those skilled in the art will recognizethat Memory/Local Persistent Data Storage 126 may include various othersoftware components, such as an operating system, various applications,other specific processes, etc.

During operation, while processing Host I/O Requests 112, Host I/OProcessing Logic 135 performs Extent Operations 152 and Non-ExtentOperations 154. Extent Operations 152 include operations performed onextents. Each extent may be a logically contiguous, multi-page portionof a shared logical address space, and may consist of and/or include atleast a portion of a data storage volume or logical drive. Eachoperation in Extent Operations 152 may be performed on one or moreextents.

In some embodiments, Extent Operations 152 include one or more copyextent operations that each create a point in time copy of a unit ofdata (e.g. create a “snapshot” of a data object). In some embodiments,Extent Operations 152 may also include one or more delete extentoperations that each delete a point in time copy of a unit of data (e.g.delete a “snapshot” of a data object).

Non-Extent Operations 154 include operations other than extentoperations. Non-Extent Operations 154 may include data write operationsthat are performed on specific pages located within specific extents.

In the example of FIG. 1 , Transaction Log Logic 138 includesTransaction Log 156, Background Operation Processing Logic 158,Classification and Selection Logic 160, and Log Data Structures 162.During operation, Transaction Log Logic 138 stores Extent Operations 152into corresponding records of Transaction Log 156. Transaction Log 156may be contained wholly or partly within the persistent data storagedevices of Memory/Local Persistent Data Storage 126. When TransactionLog Logic 138 stores each one of Extent Operations 152 into thecorresponding one of the records in Transaction Log 156, Transaction LogLogic 138 stores metadata that is sufficient for Background OperationProcessing Logic 158 to complete each operation in the correspondingrecord.

In response to each one of Extent Operations 152 being successfullystored into the corresponding record within Transaction Log 156, HostI/O Processing Logic 135 indicates successful completion of theoperation to one of the Hosts 110, e.g. by transmitting anacknowledgement to one of Hosts 110 that issued a host I/O request thatcaused the extent operation to be performed, as shown byAcknowledgements 150.

After each one of the Extent Operations 152 is successfully stored intothe corresponding record within Transaction Log 156, BackgroundOperation Processing Logic 158 performs the extent operation. AfterBackground Operation Processing Logic 158 performs an extent operation,an indication that the extent operation has been completed is stored,e.g. within a record in Transaction Log 156, and the operation is thenready to be flushed from Transaction Log 156, e.g. in parallel with oneor more other unrelated extent operations that have also been completed.

Transaction Log Logic 138 also stores Non-Extent Operations 154 intocorresponding records of Transaction Log 156. When Transaction Log Logic138 stores each one of the Non-Extent Operations 154 into thecorresponding one of the records in Transaction Log 156, Transaction LogLogic 138 stores metadata that is sufficient for Background OperationProcessing Logic 158 to complete the operation in the correspondingrecord. In response to each one of Non-Extent Operations 154 beingsuccessfully stored into the corresponding record within Transaction Log156, Host I/O Processing Logic 135 indicates successful completion ofthe operation to one of the Hosts 110, e.g. by transmitting anacknowledgement to one of Hosts 110 that issued a host I/O request thatcaused the non-extent operation to be performed, as shown byAcknowledgements 150. After each one of Non-Extent Operations 154 issuccessfully stored into the corresponding record within Transaction Log156, Background Operation Processing Logic 158 performs the non-extentoperation. After Background Operation Processing Logic 158 performs anon-extent operation, an indication that the non-extent operation hasbeen completed is stored, e.g. within a record in Transaction Log 156,and the operation is then ready to be flushed from Transaction Log 156,e.g. in parallel with one or more other non-extent and/or extentoperations that have also been completed.

Further during operation, Transaction Log Logic 138 determines thatmultiple extent operations stored Transaction Log 156 are unrelated toeach other. Such a determination may, for example, be performed byClassification and Selection Logic 160. In response to determining thatthe extent operations are unrelated to each other, after the extentoperations are completed by Background Processing Logic 158, TransactionLog Logic 138 flushes the extent operations in parallel from TransactionLog 156 to Backend Non-Volatile Data Storage Drives 128. For example,Transaction Log 156 may issue multiple parallel requests to Flush Logic154 to flush extent operations from Transaction Log 156 to BackendNon-Volatile Data Storage Drives 128, as shown by Parallel FlushRequests 162. Flush Logic 154 flushes the completed extent operationsfrom Transaction Log 156 by moving the data that is written by each ofthe completed operations from Memory/Local Persistent Data Storage 126to corresponding physical locations within Backend Non-Volatile DataStorage Drives 128, as shown in FIG. 1 by Stored User Data 130.

Classification and Selection Logic 160, in combination with TransactionLog 156 and Log Data Structures 162, maintains dependencies betweenmultiple extent operations and one or more other extent and/ornon-extent operations that are also stored in Transaction Log 156 whileflushing the extent operations in parallel from Transaction Log 156 toBackend Non-Volatile Data Storage Drives 128.

In some embodiments, Classification and Selection Logic 160, incombination with Transaction Log 156 and Log Data Structures 162,maintains the dependencies between extent operations and other extentand/or non-extent operations that are also stored in Transaction Log 156at least in part by identifying dependency chains within Transaction Log156. Each one of the dependency chains may include one or more extentoperations, and each one of the dependency chains may representdependencies between the extent operation in the dependency chain and atleast one other extent or non-extent operation stored in the TransactionLog 156.

In some embodiments, Classification and Selection Logic 160, incombination with Transaction Log 156 and Log Data Structures 162,maintains the dependencies between extent operations and other extentand/or non-extent operations that are also stored in Transaction Log 156at least in part by generating at least one tree data structure in LogData Structures 162 representing the dependencies between each of theplurality of extent operations and other operations stored in theTransaction Log 156.

In some embodiments, Classification and Selection Logic 160, incombination with Transaction Log 156 and Log Data Structures 162,maintains the dependencies between extent operations and other extentand/or non-extent operations that are also stored in Transaction Log 156at least in part by traversing at least one tree data structure in LogData Structures 162 while flushing extent operations in parallel fromthe transaction log to the Backend Non-Volatile Data Storage Drives 128in order to select the records in the Transaction Log 156 from whichoperations are to be flushed.

FIG. 2 is a block diagram showing an example of a portion of atransaction log including a dependency chain, e.g. Transaction LogPortion 200. As shown in FIG. 2 , records are added to Transaction LogPortion 200 temporally from left to right, such that records to theright are more recently added than records to the left. The TransactionLog Portion 200 of FIG. 2 contains a single example of a dependencychain spanning multiple records. A record 202 stores a write operationdirected to the page located at the offset 0x200000 within extent E1, arecord 204 stores a write operation directed to the page located at theoffset 0x240000 within extent E1, a record 206 stores an extent copyoperation that copies the contents of extent E1 (which may be eitherpointers to data or data) to extent E2, a record 208 stores a writeoperation directed to the page located at the offset 0x300000 withinextent E1, and a record 210 stores a write operation directed to thepage located at the offset 0x400000 within the extent E2. In thedependency chain shown in FIG. 2 , the write operation directed to thepage located at offset 0x200000 in the extent E1 and the write operationdirected to the page located at offset 0x240000 within extent E1 areboth flushed out of the transaction log before the copy of the extent E1to the extent E2, so that E2 after the flush of the extent copy willreflect the result of those two write operations to E1. In addition, thecopy of the extent E1 to E2 is flushed before the flushing of the writeoperation directed to the page located at offset 0x300000 in extent E1and before the flushing of the write operation directed to the pagelocated at offset 0x400000 in extent E2, so that E2 after the flush ofthe extent copy operation will not reflect either of those two writeoperations. Moreover, in the case where the extent copy of extent E1 toextent E2 creates a point in time copy of extent E1 in extent E2, extentE2 may not exist prior to the extent copy operation being performed, andaccordingly the write to the page at offset 0x400000 in extent E2 cannotbe performed and flushed until after the extent copy operation thatcopies E1 to E2 is performed and flushed.

FIG. 3 is a block diagram showing another example of a portion of atransaction log including a dependency chain, e.g. Transaction LogPortion 300. The records in Transaction Log Portion 300 are also addedtemporally from left to right, and Transaction Log Portion 300 of FIG. 3also contains a single dependency chain spanning multiple records. Arecord 302 stores a write operation directed to the page located at theoffset 0x200000 within extent E1, a record 304 stores a write operationdirected to the page located at the offset 0x240000 within extent E1, arecord 306 stores an extent copy operation that copies the contents ofextent E1 (which may be pointers to data or data) to extent E2, a record308 stores a write operation directed to the page located at the offset0x300000 within extent E1, and a record 310 stores an extent deleteoperation that deletes the extent E1. In the dependency chain shown inFIG. 3 , the write operation directed to the page located at offset0x200000 in the extent E1 and the write operation directed to the pagelocated at offset 0x240000 within extent E1 are both flushed out of thetransaction log before the copy of the extent E1 to the extent E2, sothat E2 after the flush of the extent copy operation will reflect theresult of those two write operations to E1. In addition, the copy ofextent E1 to extent E2 is flushed prior to both the write to the page atoffset 0x300000 in E1 and the deletion of E1. Further, the write to thepage at offset 0x300000 in E1 may be invalidated based on the extentdelete operation performed on E1 that is stored in the following record.

FIG. 4 is a block diagram showing an example of a portion of atransaction log including multiple dependency chains, e.g. TransactionLog Portion 400. The records in Transaction Log Portion 400 are alsoadded temporally from left to right, with new records being added at theLog Head and records being processed at the Log Tail. The TransactionLog Portion 400 may, for example, be part of a transaction log that is acircular or ring data structure. Each record in Transaction Log Portion400 uniquely corresponds to a single Log Sequence Number (LSN). The LSNsare monotonically increasing, and the records in Transaction Log Portion400 correspond to LSNs 10-21. In FIG. 4 , data write operations arerepresented in the following format:

-   -   “Extent Number”:“Number of write to same Extent”

Accordingly, in this format the data write “E1:1” is the first write toextent E1, the data write “E1:2” is the second write to extent E1, thedata write “E1:3” is the third write to extent E1, and so on. Similarly,the data write “E2:1” is the first write to extent E2, the data write“E2:2” is the second write to extent E2, and so on. Data writes to theextents E3 and E4 are represented similarly.

A first dependency chain in Transaction Log Portion 400 is made up ofthe operations stored in the records corresponding to LSN 10, LSN 11,LSN 13, LSN 14, LSN 16, LSN 18, and LSN 21. A second dependency chain inTransaction Log Portion 400 is made up of the operations stored in therecords corresponding to LSN 12, LSN 15, LSN 17, LSN 19, and LSN 20.

The extent operations in the two dependency chains shown in theTransaction Log Portion 400 are unrelated. Specifically, the extent copyoperation that copies the contents of extent E1 to extent E3 and isstored in the record corresponding to LSN 13, and the extent copyoperation that copies the contents of the extent E3 to extent E5 and isstored in the record corresponding to LSN 21, are unrelated to theextent copy operation in the second dependency chain that copies thecontents of extent E2 to extent 4 and is stored in the recordcorresponding to LSN 17. The dependencies in the two dependency chainsare therefore also unrelated. The two dependency chains may therefore beflushed in parallel by the disclosed technology. For example, the datawrites E1:1 (stored at LSN 10), E1:2 (stored at LSN 11), E2:1 (stored atLSN 12), and E2:2 (stored at LSN 15) may be flushed in parallel by thedisclosed technology. The extent copy operation stored at LSN 13 thatcopies the contents of extent E1 to extent E3, and the extent copyoperation stored at LSN 17 that copies the contents of the extent E2 toextent E4 is then flushed in parallel thereafter. The data writes E3:1(stored at LSN 14), E1:3 (stored at LSN 16), and E3:2 (stored at LSN 18)must be flushed before the extent copy operation stored at LSN 21 thatcopies the contents of extent E3 to extent E5. In addition, the datawrites E4:1 (stored at LSN 19) and E2:3 (stored at LSN 20) may beflushed in parallel along with the data writes E3:1 (stored at LSN 14),E1:3 (stored at LSN 16), and E3:2 (stored at LSN 18) or the extent copyoperation stored at LSN 21 that copies the contents of the extent E3 tothe extent E5, since the operations are not related and there is nodependency between them. While the example of FIG. 4 shows only twodependency chains, those skilled in the art will recognize that largernumbers of chains may be identified, along with extent deleteoperations, and that additional opportunities for increased parallelismmay thus be provided.

FIG. 5 is a block diagram showing an example of transaction log datastructures in some embodiments. The data structures shown in FIG. 5 maybe contained in the Log Data Structures 162 shown in FIG. 1 . The properordering of dependencies (i.e. between records in the transaction logstoring dependent operations) while providing flushing parallelism isaccomplished (e.g. through operation of Classification and SelectionLogic 160) by classifying operations stored in records in thetransaction log with additional Barrier LSNs within data structuresreferred to herein as “Buckets”. DataBucket type Buckets may be placedinto a DataBucketStore 500 and/or into an LSNTree 502. “ExtentOpBucket”type Buckets are placed into the LSNTree 502, and may contain chains ofdependent “DataBucket” type Buckets.

For example, with reference to the Transaction Log Portion 400 shown inFIG. 4 , the records storing the data writes E1:1 (the record at LSN10), E1:2 (the record at LSN 11), and E2:1 (the record at LSN 12), storeBarrier LSNs of 0, or a number less than 10 if there is a record storingan extent copy operation prior to those records. The record at LSN 16storing the extent copy operation that copies the contents of extent E1to the extent E3 stores a Barrier LSN of 11. The records that store thedata write operations E3:1 (the record at LSN 14) and E1:3 (the recordat LSN 16) store Barrier LSNs of 13, and so on.

Each Bucket defines a key for two trees, one key being an LBA (LogicalBlock Address) range of the Bucket, and the other key being a lowest LSNfor the Bucket, which may be used for tail flushing efficiency. The LBArange of each Bucket can be defined in various granularities, such as 2MB or 1 GB.

In the example of FIG. 5 , there are two Bucket types, ExtentOpBucketBuckets 504 and DataBucket Buckets 506. Further in the example of FIG. 5, LogRecordRing 508 stores the records of the transaction log, with eachrecord storing a PageDescriptor 510. Each PageDescriptor 510 storesdetails about the operation that is stored in the record (e.g. operationtype, relevant metadata needed to perform the operation, etc.), and theBarrier LSN for the record. LogRecordLinkArray 512 is an array ofLogRecordLinks 514, where each LogRecordLink 514 in LogRecordLinkArray512 corresponds to one of the records in LogRecordRing 508, and linksthe PageDescriptors 510 within a DataBucket Bucket 506. Each DataBucketBucket 506 is classified into the DataBucketStore 500 and either theLSNTree 502 or an ExtentOpBucket Bucket 504. The LBA range of eachDataBucket Bucket 506 maintains localization, while the LSNTree 502maintains flush priority for purposes of moving the log tail of thetransaction log.

The disclosed technology operates such that the following properties aremaintained:

1. A DataBucket Bucket 506 can be in the DataBucketStore 500 and/oreither in the LSNTree 502 or chained upon an ExtentOpBucket Bucket 504.

2. Each DataBucket Bucket 506 that is a direct node of the LSNTree 502can be flushed.

3. Each ExtentOpBucket Bucket 504 is always placed in the LSNTree 502and may contain a chain of DataBucket Buckets 506.

4. Insertion of an ExtentOpBucket Bucket 504 may remove existingDataBucket Buckets 506 from the DataBucketStore 500 in order to maintaindependencies during flushing. The DataBucket Buckets 506 that areremoved from the DataBucketStore 500 can be within a Bucket chain of anExtentOpBucket Bucket 504 or within the LSNTree 502.

5. All ExtentOpBucket Buckets 504 within the LSNTree 502 are evaluatedbased on whether they are ready to be flushed.

6. Upon completion of the flush of an ExtentOpBucket Bucket 504, theBuckets that were dependent upon the extent operation in theExtentOpBucket Bucket 504 are placed directly into the LSNTree 502 andare thereafter recognized as being ready to be flushed.

Classification of Records into the LSNTree and Dependencies

In the disclosed technology (e.g. by operation of Classification andSelection Logic 160), each record in the transaction log storing anextent operation is assigned a Bucket (i.e. an ExtentOpBucket Bucket)with an LSN and placed within the LSNTree. However, the ExtentOpBucketBuckets do not get put into the DataBucketStore. A BarrierLSN isprovided for each incoming PageDescriptor indicating whether there is adependency on any operation stored in another record of the transactionlog. The BarrierLSN allows the disclosed technology to determine whetheran operation stored in a record is dependent upon another operationstored in another record. The BarrierLSN may be used to look up anextent operation's record, to determine the extent operation's validity,and the corresponding ExtentOpBucket Bucket. The organization ofdependency provided by the disclosed technology is stored as a binarytree (e.g. the LSNTree), in which ExtentOpBucket Bucket nodes maycontain a chain of dependent DataBucket Buckets. FIG. 6 illustrates howembodiments of the disclosed technology may scan records in thetransaction log to classify operations stored in the records of thetransaction log into Buckets. The steps of FIG. 6 may, for example, beperformed by Classification and Selection Logic 160.

As shown in FIG. 6 , at step 600, a next record in the transaction logis scanned. Step 600 is followed by step 602, in which the disclosedtechnology checks the operation type of the record. In step 604, thedisclosed technology branches based on whether the operation stored inthe record is an extent delete operation. If so, step 604 is followed bystep 610. Otherwise, step 604 is followed by step 606.

In step 606, the disclosed technology branches based on whether theoperation stored in the record is an extent copy operation. If so, step606 is followed by step 610. Otherwise, step 606 is followed by step608.

In step 608, the disclosed technology branches based on whether theoperation stored in the record is a data write operation. If so, step608 is followed by step 614.

In step 614, the disclosed technology looks in the DataBucketStore for aBucket (i.e. a DataBucket Bucket) having an LBA range that includes theLBA to which the data write operation stored in the record is directed.For example, in some embodiments a corresponding a DataBucket Bucket maybe found using the destination or target LBA indicated by the contentsof the record and a corresponding Bucket LBA key.

In step 616, the disclosed technology branches based on whether aDataBucket Bucket was found in the DataBucketStore in which to representthe data write operation stored in the record. If so, step 616 isfollowed by step 620. Otherwise, step 616 is followed by step 618, inwhich a new DataBucket Bucket is created. Step 618 is followed by step620.

In step 620, the operation stored in the record is added to theDataBucket Bucket that was either located in the DataBucketStore at step614 or created at step 618. Step 620 is followed by step 622, in whichthe new or modified DataBucket Bucket is added to the DataBucketStore.Step 622 is followed by step 624, in which the disclosed technologydetermines whether the record has a BarrierLSN and whether the barrieris valid. For example, in some embodiments, the BarrierLSN is consideredvalid if it is not 0 and occurs after the log tail, and the barrier isdeemed to be valid if an ExtentOpBucket Bucket record is not yet markedcompleted. If the record has a BarrierLSN and the barrier is valid, thenstep 624 is followed by step 626. Otherwise, step 624 is followed bystep 626.

In step 626, the disclosed technology looks up the ExtentOpBucket Bucketto which the DataBucket Bucket is to be chained. For example, in someembodiments, at step 626 the barrier LSN may be used to locate therecord storing the extent operation in the LogRecordRing and thecorresponding ExtentOpBucket Bucket in the corresponding LogRecordLinkin the LogRecordLinkArray.

Step 626 is followed by step 628, in which the disclosed technology addsthe DataBucket Bucket to the chain of the ExtentOpBucket Bucketidentified in step 626. Step 628 is followed by step 632. In step 632,the disclosed technology determines whether all records have beenscanned. If so, step 632 is followed by step 634, where the processends. Otherwise, step 632 is followed by step 600.

In step 630, the disclosed technology adds the Bucket (either DataBucketBucket or ExtentOpBucket Bucket) as necessary to the LSNTree. Step 630is followed by step 632.

In step 610, the disclosed technology may remove one or more DataBucketBuckets that are to be chained to the ExtentOpBucket Bucket from theDataBucketStore. For example, at step 610 the disclosed technology mayexamine the record to determine its validity. If the extent operationstored in the record has not yet been flushed, the disclosed technologymay perform a lookup into the DataBucketStore to find the an appropriateDataBucket Bucket or create a new DataBucket Bucket to add to the chainof dependent Buckets of the extent operation.

In step 612, the disclosed technology may create the ExtentOpBucketBucket for the extent operation, with the DataBucket Buckets obtained instep 610 chained thereto. For example, a new ExtentOpBucket Bucket maybe created in step 612 for a record determined to be valid at step 610.If there is no BarrierLSN provided in the record, all DataBucket Bucketspertaining to the source extent of the extent operation (e.g. an extentcopy operation) are removed from the DataBucketStore to ensure that nonew data write operations are added to stale Buckets. New DataBucketBuckets pertaining to the LBA range of the source extent of an extentcopy operation may be added to the ExtentOpBucket Bucket. In someembodiments, if a valid BarrierLSN is present, a parent ExtentOpBucketBucket may be found using the BarrierLSN and the LogRecordLinkArray. Achain of dependent DataBucket Buckets within the parent ExtentOpBucketBucket may be scanned and are likewise removed from the DataBucketStore.The new ExtentOpBucket Bucket may then be added to the parentExtentOpBucket Bucket chain.

Step 612 is followed by step 630.

FIG. 7 shows an example of an LSN Tree 700, and illustrates an exampleof the disclosed classification process (e.g. as performed byClassification and Selection Logic 160), as applied to the exampleTransaction Log Portion 400 shown in FIG. 4 . As shown in FIG. 7 , thedata write operation E1:1 is placed into a DataBucket Bucket 706 thatalso describes the lowest LSN of any operation in the bucket (e.g. LSN10), the highest LSN, and an LBA key. The DataBucket Bucket 706 isplaced into the LBA tree and LSN Tree 700. Data write operation E1:2 isthen classified. In the case where E1:2 falls within the same LBA rangeas E1:1, it is also placed in DataBucket Bucket 706. However,alternatively, in the case where E1:2 does not fall into within the sameLBA range as E1:1, a new DataBucket Bucket may be created, and thenplaced into the LBA tree and the LSN Tree 700. As further shown in FIG.7 , a new DataBucket Bucket 702 is also created for the data writeoperations E2:1 and E2:2 in the same manner.

As further shown in FIG. 7 , an ExtentOpBucket Bucket 708 is created forthe extent copy operation E1:E3. ExtentOpBucket Bucket 708 is not placedinto the LBA tree, but is placed into the LSN Tree 700. The data writeoperation E3:1 is placed into a new DataBucket Bucket 709 that is placedinto the LBA tree. Because E3:1 has a BarrierLSN of 13 (corresponding tothe ring storing the extent copy operation E1:E3), it is placed in thedependency chain 711 that contains ExtentOpBucket 708, since it dependson the extent copy operation E1:E3 being flushed before it can beprocessed. DataBucket Bucket 709 is therefore not placed directly intothe LSN Tree 700.

Selection and Flush

During selection of operations to be flushed and the flushing of thoseoperations (e.g. as performed by Classification and Selection Logic 160based on the LSN Tree 700), the placement of DataBucket Buckets by thedisclosed technology directly within the LSN Tree 700 indicates thatthere is no dependency and that the operations they contain can beflushed in parallel. During the selection process, Buckets that areplaced in the LSN Tree 700 (e.g. Buckets 701, 702, 704, 706 and 708 inFIG. 7 ) undergo evaluation based on their BarrierLSN and the log tail.Buckets in the LSN Tree 700 are selected for flushing in LSN order tomaintain priority of log tail movement. When an extent operation isflushed from a record, a “Done” record may be created for persistence inthe transaction log through a commit sequence, in order to indicatecompletion of the extent operation. The corresponding “Done” record isprocessed on the commit of the record and the original record may thenbe invalidated. DataBucket Buckets in the chain of an ExtentOpBucketBucket for the extent operation may then be promoted into direct nodesof the LSN Tree 700 to be flushed. FIGS. 8 and 9 illustrate how thecompletion of an extent copy operation (e.g. extent copy E1:E3) resultsin the removal of the ExtentOpBucket Bucket 708 from LSN Tree 700 andthe promotion of the dependent DataBucket Buckets 709 and 710 so thatthey are stored directly within the LSN Tree 700.

FIG. 8 is a block diagram showing another example of the LSN Tree 700.As shown in FIG. 8 , the data write operations E1:1 and E1:2 inDataBucket Bucket 706, and the data write operations E2:1 and E2:2 inDataBucket Bucket 701, have previously been flushed. Accordingly, theresulting LSN Tree 700 shown in FIG. 8 contains the ExtentOpBucketBuckets 700, 708, and 704. Since Buckets directly contained in the LSNTree 700 are selected for flushing in order of their LSNs, withconsideration of their BarrierLSNs, and the BarrierLSN forExtentOpBucket Bucket 708 is LSN 11 indicating the record storing thedata write E1:2, and since DataBucket Bucket 706 has now been flushed,the next extent operation selected for flushing in the example of FIG. 8is the extent copy E1:E3 in ExtentOpBucket Bucket 708. It should berecognized that because the extent copy operations E1:E3 and E2:E4 areunrelated, that the BarrierLSN for ExtentOpBucket Bucket 701 is LSN 13,and that DataBucket Bucket 702 has now been flushed. Accordingly, theextent copy operation E2:E4 could also be flushed in parallel with theextent copy operation E1:E3. And it should also be recognized that theextent copy operation E3:E5 has a BarrierLSN of LSN 18, indicating thedata write operation E3:2, which cannot be flushed until after theextent copy operation E1:E3 is flushed, and that accordingly the extentcopy operation E3:E5 cannot be flushed in parallel with the extent copyoperation E1:E3.

FIG. 9 is a block diagram showing a further example of the LSN Tree 700after flushing of the extent copy operation E1:E3, resulting in theremoval of ExtentOpBucket Bucket 708 from the LSN Tree 700, resulting inthe promotion of the DataBucket Buckets 709 and 710 to Buckets that aredirectly contained in the LSN Tree 700. In the LSN Tree 700 as shown inFIG. 9 , the data write operations in DataBucket Buckets 709 and 710,and the extent copy operations in ExtentOpBucket Buckets 700 and 704 canall be selected for flushing in parallel.

As will be appreciated by one skilled in the art, aspects of thetechnologies disclosed herein may be embodied as a system, method orcomputer program product. Accordingly, each specific aspect of thepresent disclosure may be embodied using hardware, software (includingfirmware, resident software, micro-code, etc.) or a combination ofsoftware and hardware. Furthermore, aspects of the technologiesdisclosed herein may take the form of a computer program productembodied in one or more non-transitory computer readable storagemedium(s) having computer readable program code stored thereon forcausing a processor and/or computer system to carry out those aspects ofthe present disclosure.

Any combination of one or more computer readable storage medium(s) maybe utilized. The computer readable storage medium may be, for example,but not limited to, a portable computer diskette, a hard disk, a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), a portable compact discread-only memory (CD-ROM), an optical storage device, a magnetic storagedevice, or any suitable combination of the foregoing. In the context ofthis document, a computer readable storage medium may be anynon-transitory tangible medium that can contain, or store a program foruse by or in connection with an instruction execution system, apparatus,or device.

The figures include block diagram and flowchart illustrations ofmethods, apparatus(s) and computer program products according to one ormore embodiments of the invention. It will be understood that each blockin such figures, and combinations of these blocks, can be implemented bycomputer program instructions. These computer program instructions maybe executed on processing circuitry to form specialized hardware. Thesecomputer program instructions may further be loaded onto programmabledata processing apparatus to produce a machine, such that theinstructions which execute on the programmable data processing apparatuscreate means for implementing the functions specified in the block orblocks. These computer program instructions may also be stored in acomputer-readable memory that can direct a programmable data processingapparatus to function in a particular manner, such that the instructionsstored in the computer-readable memory produce an article of manufactureincluding instruction means which implement the function specified inthe block or blocks. The computer program instructions may also beloaded onto a programmable data processing apparatus to cause a seriesof operational steps to be performed on the programmable apparatus toproduce a computer implemented process such that the instructions whichexecute on the programmable apparatus provide steps for implementing thefunctions specified in the block or blocks.

Those skilled in the art should also readily appreciate that programsdefining the functions of the present invention can be delivered to acomputer in many forms; including, but not limited to: (a) informationpermanently stored on non-writable storage media (e.g. read only memorydevices within a computer such as ROM or CD-ROM disks readable by acomputer PO attachment); or (b) information alterably stored on writablestorage media (e.g. floppy disks and hard drives).

While the invention is described through the above exemplaryembodiments, it will be understood by those of ordinary skill in the artthat modification to and variation of the illustrated embodiments may bemade without departing from the inventive concepts herein disclosed.

What is claimed is:
 1. A method comprising: storing a plurality ofextent operations into corresponding records of a transaction log; inresponse to successfully storing the plurality of extent operations intothe corresponding records of the transaction log, indicating successfulcompletion of each of the plurality of extent operations to at least onehost; determining that the plurality of extent operations are unrelatedto each other; and in response to determining that the plurality ofextent operations are unrelated to each other, flushing the plurality ofextent operations in parallel from the transaction log to back-endnon-volatile data storage; wherein the flushing of the plurality ofextent operations in parallel from the transaction log to back-endnon-volatile data storage further comprises maintaining dependenciesbetween the plurality of extent operations and other operations storedin the transaction log.
 2. The method of claim 1, further comprising:wherein maintaining the dependencies between the plurality of extentoperations and other operations stored in the transaction log furthercomprises identifying a plurality of dependency chains within thetransaction log, wherein each one of the dependency chains includes oneof the plurality of extent operations, and wherein each one of thedependency chains represents dependencies between the one of the extentoperations in the dependency chain and at least one other operationstored in the transaction log.
 3. The method of claim 2, whereinmaintaining the dependencies between the plurality of extent operationsand the other operations stored in the transaction log further comprisesgenerating at least one tree data structure representing thedependencies between each of the plurality of extent operations and theother operations stored in the transaction log.
 4. The method of claim3, wherein maintaining the dependencies between the plurality of extentoperations and the other operations stored in the transaction logfurther comprises traversing the at least one tree data structure whileflushing the plurality of extent operations in parallel from thetransaction log to back-end non-volatile data storage to select recordsin the transaction log to flush.
 5. The method of claim 1, wherein theplurality of extent operations includes at least one extent copyoperation.
 6. The method of claim 1, wherein the plurality of extentoperations includes at least one extent delete operation.
 7. A datastorage system comprising: at least one storage processor includingprocessing circuitry and a memory; a plurality of non-volatile datastorage drives communicably coupled to the at least one storageprocessor; and wherein the memory has program code stored thereon,wherein the program code, when executed by the processing circuitry,causes the processing circuitry to: store a plurality of extentoperations into corresponding records of a transaction log; in responseto successfully storing the plurality of extent operations into thecorresponding records of the transaction log, indicate successfulcompletion of each of the plurality of extent operations to at least onehost; determine that the plurality of extent operations are unrelated toeach other; and in response to a determination that the plurality ofextent operations are unrelated to each other, flush the plurality ofextent operations in parallel from the transaction log to back-endnon-volatile data storage; wherein the flush of the plurality of extentoperations in parallel from the transaction log to back-end non-volatiledata storage further comprises causing the processing circuitry tomaintain dependencies between the plurality of extent operations andother operations stored in the transaction log.
 8. The data storagesystem of claim 7, wherein the program code, when executed by theprocessing circuitry, further causes the processing circuitry tomaintain the dependencies between the plurality of extent operations andother operations stored in the transaction log at least in part bycausing the processing circuitry to identify a plurality of dependencychains within the transaction log, wherein each one of the dependencychains includes one of the plurality of extent operations, and whereineach one of the dependency chains represents dependencies between theone of the extent operations in the dependency chain and at least oneother operation stored in the transaction log.
 9. The data storagesystem of claim 8, wherein the program code, when executed by theprocessing circuitry, further causes the processing circuitry tomaintain the dependencies between the plurality of extent operations andthe other operations stored in the transaction log at least in part bycausing the processing circuitry to generate at least one tree datastructure representing the dependencies between each of the plurality ofextent operations and the other operations stored in the transactionlog.
 10. The data storage system of claim 9, wherein the program code,when executed by the processing circuitry, further causes the processingcircuitry to maintain the dependencies between the plurality of extentoperations and the other operations stored in the transaction log atleast in part by causing the processing circuitry to traverse the atleast one tree data structure while flushing the plurality of extentoperations in parallel from the transaction log to back-end non-volatiledata storage to select records in the transaction log to flush.
 11. Thedata storage system of claim 7, wherein the plurality of extentoperations includes at least one extent copy operation.
 12. The datastorage system of claim 7, wherein the plurality of extent operationsincludes at least one extent delete operation.
 13. A computer programproduct including a non-transitory computer readable medium havinginstructions stored thereon, wherein the instructions, when executed onprocessing circuitry, cause the processing circuitry to perform thesteps of: storing a plurality of extent operations into correspondingrecords of a transaction log; in response to successfully storing theplurality of extent operations into the corresponding records of thetransaction log, indicating successful completion of each of theplurality of extent operations to at least one host; determining thatthe plurality of extent operations are unrelated to each other; and inresponse to determining that the plurality of extent operations areunrelated to each other, flushing the plurality of extent operations inparallel from the transaction log to back-end non-volatile data storage;wherein the flushing of the plurality of extent operations in parallelfrom the transaction log to back-end non-volatile data storage furthercomprises maintaining dependencies between the plurality of extentoperations and other operations stored in the transaction log.